On-chip spectral filtering using ccd array for imaging and spectroscopy

ABSTRACT

A spectral filtering apparatus and method is presented, which enables the functions of hyperspectral imaging to be performed with increased speed, simplicity, and performance that are required for commercial products. The apparatus includes a photosensitive array having a plurality of photosensitive elements such that different subsections of photosensitive elements of the array receive light of a different wavelength range of a characteristic spectrum of a target, and output electronics combines signals of at least two photosensitive elements in a subsection of the array and outputs the combined signal as a measure of the optical energy within a bandwidth of interest of the characteristic spectrum.

FIELD OF THE INVENTION

The present invention relates generally to imaging and spectralanalysis, and more particularly relates to apparatus and methods forimaging, data acquisition, filtering and processing.

BACKGROUND OF THE INVENTION

Various imaging, scanning, and detection techniques for detecting andquantifying fluorescent labels are known in the art. These techniquesdiffer in their detection capability, speed, and cost, but a commonchallenge to all fluorescence imaging techniques is the separation ofthe light used to excite the targeted fluorophores from the emittedfluorescence light. One common method uses a combination of dichroicfilters/beamsplitters and band-pass filters that isolate thefluorescence light for detection. This approach is characterized bylimitations as to the number of separate fluorescence emissions and thenumber of detection channels that can be used in the same system inparallel. Significantly, this approach requires fixed band-pass filtersand thus cannot be easily changed to adapt to variations in thewavelength(s) of the fluorescence light being detected.

Another approach is to use a tunable band-pass filter, either betweenthe emitted fluorescence and the detector, or in front of theilluminating source. For example, U.S. Pat. No. 5,379,065 discloses aspectral imaging device for use on a space vehicle to acquireterrestrial surface images, wherein a spectrally agile filter (SAF) isused to rapidly change the spectral pass band to acquire multiple imagesthrough different spectral filters; U.S. Pat. No. 6,690,466 discloses aspectral illuminator that is controlled to selectively emit light ineach of a number of independent wavelength bands. The type of filterused and the tuning method depends on the speed of tunability, insertionloss, and whether the imaging method is a point imager or an areaimager. The tunable band-pass filter approach falls into themulti-spectral class if the operating spectral resolution is coarse(e.g., on the order of tens of nanometers) and into the hyperspectralclass if it has a much higher spectral resolution (e.g., in thesub-nanometer range) U.S. Pat. Nos. 6,495,363 and 6,495,818 provideexamples of hyperspectral filtering. In the prior art hyperspectralmethods, the data is processed post image acquisition.

The tunable band-pass filter approach requires the use of at least onetunable filter per detection channel, and measurements need to be takenat each spectral position (i.e., wavelength band) sequentially. Unlessthe technology used in tuning the filter is fast, this approach tends tobe slow, particularly if higher spectral resolution is needed. Of thevarious tunable filters implemented in fluorescence filtering, thefastest are Liquid Crystal (LC) filters as disclosed in the '466 patent,and Acousto-Optic filters (AOTF) as disclosed in the '065 patent. Thesefilters are fast, but they are expensive and suffer from high opticalinsertion loss. Furthermore, since they require sequential detectionband-by-band, they can result in a much slower process than the filteritself is capable of. Another type of device that can be used as atunable filter is the spectrometer. This device can perform the functionof tuning the wavelength being detected, but at a much slower speedsince the mode of tuning is typically mechanical. Spectrometers also areexpensive and have an even higher insertion loss than the LC or AOTFfilters.

Yet another approach in fluorescence detection is the use of spectrallydispersive elements, such as gratings and prisms, to spread the spectralcontent of the collected light across an array detector. The desiredspectral resolution and the method of imaging dictate the type ofdispersive element to use. Similar to the tunable filter approach, thedispersive element approach can fall either into the multi-spectralclass or the hyperspectral class depending on its targeted spectralresolution. This method further requires the use of some type of arraydetector. It typically uses either a linear array detector with pointimaging, or an area array detector with line imaging. In both cases, onedimension of the array detector is used for wavelength distribution. Forthis reason, an image is acquired for one point or one line at a time.The illumination/detection device is then scanned across the target inorder to build the whole two-dimensional image. An array of spectraldata is acquired for each imaged point/line and stored in a hostcomputer. The spectral filtering of the data is processed after the scanis finished. In this manner the data is available for application ofvarious schemes of filtering, and therefore the data processing can beoptimized for the desired function at hand. The dispersive elementapproach thus offers significant flexibility as compared to the fixedfilter and the tunable filter approaches. However, because a significantamount of data is read and stored for each point or line, the speed ofoperation and the storage capacity required can become overwhelming,even for a small area scan. This has been one of the main reasons thatthis approach has not moved into commercialization.

As an example of the storage requirements for a hyperspectral operationwith post-acquisition processing, we consider the case of scanning asingle microscope slide (25 mm×75 mm) with 5 nm spatial resolution and 5nm spectral resolution across a 400 nm spectral range. Assuming thatline illumination is used, the entire 25 mm width of the slide is imagedat once, and the line is scanned across the 75 mm length of the slide.This means that 5000 image pixels are needed for the line in order toobtain the 5 μm spatial resolution (25 mm/5 μm) across the width, andthat 15,000 lines must be scanned in order to obtain the 5 μm spatialresolution (75 mm/5 μm) across the length. So, a frame of 5000×(400 nm/5nm)=400×10³ pixels are read every line or 6×10⁹ pixels for a microscopeslide. With a 12 bit A/D conversion, this means that 9 gigabytes ofstorage capacity would be needed for the scan data of a single standardsize microscope slide.

Consequently, there exists a need to reduce the amount of dataprocessing and storage requirements and thereby the required speed ofscanning and processing operations, in order to benefit from theflexibility of hyperspectral imaging. The present invention offers apowerful way to harness such filtering flexibilities with only minimaldata manipulations and substantially reduced storage capacityrequirements.

SUMMARY OF THE INVENTION

The present invention overcomes the restrictions of the prior art andprovides a significant advance in the art. An important conceptunderlying the invention is that the processing “smartness” that isusually performed after imaging data acquisition is built into the arraydetector itself. In other words, the array detector itself is programmedto perform filtering operations on-chip or within the electroniccircuitry that interfaces directly with the array detector.

According to one aspect of the invention, A spectral filter is provided,including a photosensitive array having a plurality of photosensitiveelements, at least a subsection of photosensitive elements of the arrayalong one direction thereof being configured such that each element ofthat subsection along the one direction receives light of a differentwavelength range of a characteristic spectrum and produces an electricalsignal corresponding thereto,

with the subsection being configured to combine signals of at least twophotosensitive elements and to output the combined signal as a measureof the optical energy within a bandwidth of interest of thecharacteristic spectrum.

According to another aspect of the invention, a method of performingspectral filtering using a photosensitive array having a plurality ofphotosensitive elements includes the steps of calibrating the array suchthat each element of the photosensitive array along one directionthereof corresponds to a different wavelength range of a characteristicspectrum projected onto the array; and configuring a read-out process ofthe array to combine signals of at least two photosensitive elementsalong said one direction, generated in response to illumination by thecharacteristic spectrum, as a measure of the optical energy within abandwidth of interest of said spectrum.

The above and/or other aspects, features and/or advantages of variousembodiments will be further appreciated in view of the followingdetailed description in conjunction with the accompanying figures.Various embodiments can include and/or exclude different aspects,features and/or advantages where applicable. In addition, variousembodiments can combine one or more aspect or feature of otherembodiments where applicable. The descriptions of aspects, featuresand/or advantages of particular embodiments should not be construed aslimiting other embodiments or the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The various embodiments of the present invention are shown in theaccompanying figures by way of example and not limitation, in which:

FIG. 1A is a diagram of a CCD array that is usable with the concepts ofthe present invention;

FIG. 1B is a diagram of a CCD array illustrating hyperspectralline-imaging according to one embodiment of the invention;

FIGS. 2A-2C are diagrams illustrating operation of a CCD array as aspectral filter for a single band of interest according to oneembodiment of the invention;

FIGS. 3A and 3B are diagrams illustrating operation of a CCD array as aspectral filter for multiple bands of interest according to oneembodiment of the invention;

FIG. 4 is a diagram of an imaging system according to one embodiment ofthe invention;

FIG. 5 is a chart showing quantum efficiency of a CCD usable with thepresent invention;

FIG. 6 is a diagram of imaging and timing electronics circuit system inaccordance with an embodiment of the invention;

FIG. 7 is a diagram showing imaging considerations in accordance withthe invention;

FIG. 8 is a diagram of exemplary imaging optics of the system of FIG. 4;

FIG. 9 is a diagram of an exemplary illumination source of the system ofFIG. 4;

FIG. 10 is a chart of measured illumination uniformity of the source ofFIG. 9;

FIG. 11 is a diagram of another exemplary illumination source of thesystem of FIG. 4;

FIGS. 12 and 13 are charts show a measurement of the uniformity,line-width, and flux densities achieved with the laser illuminationlight source of FIG. 11;

FIG. 14 is a diagram showing angular illumination of a target to beimaged according to another aspect of the invention;

FIG. 15 shows an exemplary dispersive element of the system of FIG. 4;

FIG. 16 is a chart of diffraction efficiency of the dispersive elementof FIG. 15; and

FIGS. 17A and 17B are diagrams showing an example of the imagingresolution capability according to the invention.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

According to one embodiment of the invention, a CCD (charge-coupleddevice) array detector is used for hyperspectral imaging and filtering.A CCD array consists of a matrix of pixels that are sensitive to light.In response to the impingement of light photons, individual pixels inthe array generate a charge of electrons, the amount of which varies inproportion to the magnitude of light photons interacting with the pixelareas and proportional to exposure time. The resulting electricalsignals are read out of the array and are interpreted to correspond tothe amount of light that generated the electrical charges in the pixellayers. The most common CCD architectures used in the instrumentationindustry today are Full-Frame, Frame Transfer, and Interline(progressive) Transfer. In the former two, the pixels are typicallyconstituted by photodiodes and the generated charges from the pixels ofthe entire array are read out directly or transferred simultaneously toanother storage array on the chip from which they are then read out. Onthe other hand, the progressive type, as shown in FIG. 1A, consists ofpixels that perform a charge transfer function in addition to beingphotosensitive. The generated charges are transferred vertically line byline to an output stage, and are subsequently read out horizontallypixel by pixel. The interline transfer type of CCD array offers a numberof advantages over its full-frame counterpart, particularly whensub-framing and read-out manipulations are needed. The followingdetailed description is thus based on the use of an interline transfertype CCD array for purposes of explanation; however, the concepts of theinvention are not limited to the use of interline transfer CCD arrays assimilar schemes can be easily devised by those skilled in the art fromthis detailed description to work for linear and full-frame area arrayCCDs.

When such a CCD array detector is used to image a two-dimensional scene,for example when used in a video capturing device, each of the pixelsrepresents a separate point in space and therefore the electrical signalread from each pixel is mapped to represent an amount of light comingfrom a corresponding point in space. On the other hand, in a typicalhyperspectral imaging configuration, the light from a single line inspace is spread out over multiple detector pixel lines as indicated inFIG. 1B. In other words, the spectral content of the line image isprojected onto the entire pixel area of the detector.

According to a key concept of the invention, instead of reading out thesignals from all of the pixels and then manipulating and processing theread out data to obtain information of a desired spectral rangepost-acquisition, the array detector is programmed to select the desiredrange of pixels that correspond to a desired wavelength range to bemeasured, add their electrical signals together before being read out,and then read out only the summed signal. This method according to theinvention saves a significant amount of time and processing power, andalso substantially reduces the amount of storage capacity needed forstoring and processing post-acquisition data. Two examples will now begiven of specific implementations of the invention.

1. On-Chip Binning

When the line image is spectrally spread across the CCD array with aknown nm/row spectral resolution (or nm/pixel spectral resolution for aline sensor imaging a point image), the specific rows in the array thatcorrespond to a desired spectral filter can be easily identified andtheir charges binned together, i.e., their signals added together,before the data is read out of the detector. In this manner, only onerow of pixels needs to be read out for each desired filter instead ofeach of the rows individually that constitute the filter, and thus theentire detection and data processing operation is considerablysimplified.

Furthermore, binning pixels together before they are read out results ina significant improvement in the signal-to-noise ratio. Morespecifically, weak fluorescence signals that could be within the levelof noise are added together before they are read out, and therefore havea significantly improved chance of being detected than if they were readout individually and added together later, as low level signals would befiltered out as noise and not stored. This results in an improved levelof detection sensitivity, without any improvement in detector materials.The dynamic range of the total spectral window is however limited to thedynamic range of the analog-to-digital converter (ADC), because there isno post-acquisition summation.

2. Readout Binning

The “binning” function also can be performed by the read-outelectronics. For example, through the use of a Digital Signal Processor(DSP) or Field-Programmable Gate Array (FPGA), one can program thereadout electronics to add or “bin” together signals from the desiredspectral range before they are fed to a host computer or otherprocessing circuit. The summation of the entire spectral window resultsin an increase in the dynamic range, and since readout noise istypically a large component of the total pixel noise at a high framerate, the total pixel noise is also increased because of the summationafter the values have been read out of the CCD chip. Fortunately, thesignals add more quickly than the noise, resulting in an increasedsignal-to-noise ratio. This method of binning thus can build the totaldynamic range beyond that of the A/D converter.

Two examples of readout binning according to the invention are shown inFIGS. 2A-2C and FIGS. 3A-3B. Where the desired spectral range is asingle band of contiguous wavelengths as shown in FIG. 2A, each row ofpixels in the desired band is binned or added together with the otherrows in the band, to form a single line or row of pixels, while theremaining rows of pixels are simply discarded by being subjected to a“fast dump” or similar operation as shown in FIG. 2B, wherein theaccumulated charge is quickly dissipated by coupling the unwanted pixelrows to ground or equivalent operation. The single line or row of pixelsrepresenting the summed data is then read out of the CCD chippixel-by-pixel as shown in FIG. 2C.

FIG. 3A shows an example where multiple spectral bands 1-4 are ofinterest. The rows of pixels within each band are summed or binnedtogether to form a single line of pixels corresponding to the desiredband, and the remaining pixel data is subjected to a fast dump operationand discarded as shown in FIG. 3B. Then, each single row of summedpixels corresponding to each of the bands 1-4 of interest are read outof the CCD chip in a pixel-by-pixel fashion.

The above-introduced spectral filtering technique can be applied tovarious multi-spectral and hyperspectral imaging methods that use arraydetectors. The invention contemplates that the technique will use aconfiguration that takes into account the desired feature requirementslisted above. In this way, the results can be easily applied to developa microarray imaging system with comparable capabilities.

A discussion of the considerations involved in choosing an opticalsystem architecture will now be presented, followed by a discussion ofvarious options that may be used for each of the sub-systems and adescription of a specific illustrative system configuration according toan embodiment of the invention.

FIG. 4 shows a diagram of a basic configuration for an optical systemwith hyperspectral/multi-spectral detection capability. It consists ofan illumination source 401, imaging optics 402, a dispersive element403, and an array detector 404. The choice of illumination source,imaging optic method, and type of array detector depend on theexcitation wavelengths used, the desired imaging resolution, the size ofthe imaging area, the speed of image acquisition required, and costconsiderations.

Imaging a target 405 microscope slide (25 mm×75 mm) at 5 μm spatialresolution rules out point imaging with mechanical scanning, such as inknown point imaging products that use optical microscopes and variousoptical filters; imaging such a slide with a point source imagingapparatus would take hours, even though the point imaging produces ahigh illumination flux density and therefore good signal-to-noiseperformance. One can achieve faster acquisition times by replacing themechanical scanning mechanism with one or more optical scanners, but ata much higher cost. Faster imaging also can be achieved using area CCDimaging, which is a method adopted by many microarray scanners. However,this approach requires the area CCD detector to be cooled considerablybecause a long CCD integration time on the order of tens of seconds perframe is required, as the illumination flux density that can be producedfor area imaging is low. Further, the relatively long integration timescause different frames captured at different locations to have differentlevels of brightness, even for the same amount of fluorophore. This isknown as a “tiling mismatch.” A significant amount of effort has beenmade in the field in attempting to reduce this effect, but to no greatsuccess.

A better approach for imaging such a target is a balance betweenpoint-imaging and area imaging, such as line imaging withone-dimensional mechanical scanning. Imaging a line instead of an areameans that all the illuminating light can be concentrated to produce ahigher flux density and therefore relatively short integration times andonly moderate detector cooling are needed. Furthermore, one can optimizethe size of the imaged line to simplify the mechanical scanning needsand still achieve the other requirements. In particular, the width of aslide (25 mm) can be imaged with 5 μm resolution using a 5000 pixel longline array detector. Mechanical scanning thus is needed only along thelength of the slide and frame tiling is avoided. The width of theillumination line preferably is matched to the desired resolution in thescanning direction, e.g., 5 μm in the described preferred embodiment.Doing so avoids the use of slit apertures as required for hyperspectraldispersion detection and which is often adopted by many prior artspectroscopic imaging solutions, such as in U.S. Pat. Nos. 6,495,818,6,495,363, and 6,245,507. “Narrow” line illumination can be achieved byprojecting a line with a diffraction-limited width; the latter can beproduced through the use of diffraction-limited light sources,projecting slit apertures placed in the illumination path, or similarmeans.

Detector

The architecture of a system according to the invention can be designedby first selecting a CCD detector with a number of pixels close to 5000and having good performance specifications. Imaging optics can be thenselected to project the width of the target slide onto the CCD arraywith good chromatic correction and low distortion. A laser excitationsource can be built to generate a matching illumination line, and anappropriate dispersive element can be selected to produce nm-scalespectral resolution over the CCD array surface. The various optionsavailable for these sub-systems and related measurements will now bediscussed.

There are a number of features that are desired for a good CCD solutionfor a spectral filter as contemplated by the invention. Some of theseare

“large” format, ideally 5000-pixel long or more

good Quantum Efficiency (QE), particularly in the near IR (700 nm-850nm) range

good sensitivity and low-dark noise

fast readout rate

fast dump capability in order to obtain high frame rates by dumping theportions of the array that are not needed

fast electronic shuttering in order to precisely control the amount ofintegration

anti-blooming

reliable supply source

One example of a CCD that matches the above requirements is theKAI-11000M manufactured by Eastman Kodak Company. CCD image sensor chipsmanufactured by Dalsa Semiconductor have been found to have similarspecifications. Other commercially available chips with varyingspecifications also can be adapted to work in accordance with theconcepts of the present invention, with some trade-offs in resolution orimage size perhaps being necessary.

The specifications for the KAI-11000M CCD chip are listed in Table 1.Most interline CCD chips suffer from a reduced quantum efficiency (QE)performance as compared to full-frame transfer CCDs, because a portionof each pixel of the interline type CCD is used for storing charges tobe transferred to adjacent pixels, and therefore not all of the pixelarea is available for interaction with light photons. However, theKAI-11000M chip was found to have an architecture that uses a microlenson top of each pixel to enhance its collection efficiency. The quantumefficiency consequently is comparable to full-frame formats, whilemaintaining all the other advantages of the interline structure. TheKAI-11000M QE is on the order of 12% at 800 nm, as shown in FIG. 5.While higher quantum efficiencies, including in the near-IR, areavailable with other CCD chips, they are smaller in size and thereforemay not be acceptable for the desired scanning simplification accordingto the invention. TABLE 1 Parameter Value Architecture Interline CCD;Progressive Scan Total Number of Pixels 4072 (H) × 2720 (V) = approx.11.1 M Number of Effective Pixels 4032 (H) × 2688 (V) = approx. 10.8 MNumber of Active Pixels 4008 (H) × 2672 (V) = approx. 10.7 M Number ofOutputs 1 or 2 Pixel Size 9.0 μm (H) × 9.0 μm (V) Imager Size 43.3 mm(diagonal) Chip Size 37.25 mm (H) × 25.70 mm (V) Aspect Ratio 3:2Saturation Signal 60,000 electrons Quantum Efficiency 0.32, 0.27, 0.25Output Sensitivity 13 μV/e Total Noise 30 electrons Dark Current <50mV/s Dark Current Doubling 7° C. Temperature Dynamic Range 66 dB ChargeTransfer Efficiency <0.99999 Blooming Suppression >1000 X Smear <−80 dBImage Lag <10 electrons Maximum Data Rate 28 MHz Package 40-pin, CerDIP,0.070″ pin spacing Cover Glass IR Cutoff or AR Coated

Evaluation electronics boards that allow for modification of the mannerin which data is read out of the CCD chip are known in the art and arecommercially available from various companies including Eastman Kodak.One such system as shown in FIG. 6 consists of a CCD chip mounted on animaging board that is connected to a timing generator board. The timinggenerator board in turn is connected to an image acquisition cardresiding in a host computer. A timing board can use a ComplexProgrammable Logic Device (CPLD), such as the 7000S ISP PLD sold byAltera Corporation, to generate the required clock signals from a 60 MHzsystem clock. The various CCD functions performed according to theinvention, including timing controls, fast-dumping, electronicshuttering, binning, etc. can be accomplished by programming thisdevice.

Imaging

The use of a 4000-pixel long CCD means that only 20 mm-long lines can beimaged with 5 μm resolution, i.e. 4000×5 μm=20 mm. The length of theKAI-11000M CCD is ˜4000×9 cm=36 mm. As indicated in FIG. 7, the task isto project (image) a 20 mm-long line onto the CCD array such that itcovers the entire 36 mm side of the CCD with chromatic correction acrossthe visible and near IR ranges (i.e., 500 nm to 900 nm), with a goodcollection efficiency and minimal distortion. This means that theimaging system will be operating at a magnification M (object-to-CCD) of$M = {\frac{36{mm}}{20{mm}} \approx {1.8x}}$

One option is to use a single lens reflex (SLR) camera-type lens that isdesigned for at least 35 mm film/CCD formats. This size presents morechallenges than 0.5″, 2/3″, and 1″ formats and therefore only higher-endlenses can achieve good color and distortion corrections. Furthermore,it is desired to use a lens with a low F-stop number in order tomaximize the collection efficiency. This narrows the choices to lenseswithin the class of Macro-imaging. A number of lenses designed andmarketed for consumer macro-imaging applications are known andcommercially available for this purpose. While not inexpensive, suchlenses tend to be much more economical than designing a special lens forthe particular application.

A number of lenses of focal lengths ranging from 25 mm to 75 mm withF-stops in the range of 1.8 to 2.8 were tested in accordance with theobjectives of the invention. The AF Micro-Nikkor 60 mm 1:2.8 D availablefrom Nikon Corporation showed better overall performance than otherlenses. The lens is designed for a 35 mm detector size and to operatewith the object being at the longer working distance side of the lens.However, since the lens is symmetric, it can be used in reverse so thatthe 20 mm line dimension is projected onto the larger 36 mm CCD (seeFIG. 8). While this lens works well, it is built with certain automationfeatures which make its size larger than it could otherwise be. Asimilar lens without the automation features and thus smaller would bepreferable for some applications contemplated by the present invention.

Illumination (Excitation) Source

In order to image a 20 mm-long line target with 5 μm resolution andmaximum illumination flux density for each pixel point, the optimalillumination light source needs to generate a 20 mm×5 μm uniform lineand project it onto the width of the microscope slide. However, in orderto limit the solution to illumination sources within reasonable costranges, the uniformity requirement must be relaxed somewhat. A target of80% was set for the uniformity of line illumination in accordance withthe invention. This is a reasonable parameter for the illustrativeapplication disclosed here and can be calibrated out without any majordownside tradeoff.

Two main variables to be considered in designing and using such lineillumination generators are 1) wavelength and type of light sources and2) angle of incidence. The first variable depends on the wavelength ofinterest and available sources for generating such wavelength, while thesecond variable depends on the type of target sample that can be usedwith the system. Two types of light sources were tested in accordancewith the invention: a white light source and two-wavelength laser diodes(685 nm and 785 nm.) A brief description of these solutions and therelated test results are presented below.

White Light

A 150 W halogen lamp source with a line array fiber optic coupling wasselected as a white light source, as shown in FIG. 9. The output side ofthe optical fiber is ˜38 mm by 0.25 mm. This was used in combinationwith a focusing lens and 10 μm wide optical slits to generate thedesired line. The measured uniformity with this set-up was about 80%and, using a band-pass filter of 10 nm bandwidth, the flux density wasabout 1 nW/μm2, or equivalently, 0.025 μW/pixel. A graph of the measureduniformity is presented in FIG. 10. The low flux density necessitatedlonger CCD integration times than desired and therefore thisconfiguration was not considered optimal. Higher flux densities aredesirable and therefore parameters of the configuration could be variedto increase the illumination flux density. For example, a higher wattagelight source could be used in combination with higher-temperature opticfiber, or alternatively parabolic focusing mechanisms could be used tofurther concentrate the illumination light to produce a higher fluxdensity.

Two-Wavelength Laser Source

Another illumination line generator was built using two laser diodes(685 nm and 785 nm). These wavelengths match commercially available 700nm and 800 nm dyes. The lasers used in this set-up were the Hitachi HL6738MG, λ=690 nm, 35 mW, and the QPhotonics QLD-780-1000S, λ=780 nm, 90mW. FIG. 11 shows a diagram of the optics used in building theillumination line generator tested in accordance with the invention. Inthis configuration, other lasers can be added as needed. FIGS. 12 and 13show a measurement of the uniformity, line-width, and flux densitiesachieved with the laser illumination light source. The uniformity andline-width are measured by directly focusing the generated line onto theKAI-11000 CCD and the flux densities are measured using an optical powermeter placed at the focused line.

Co-Axial vs. Angular Illumination Orientation

As shown in FIG. 14, illuminating the sample at an angle reduces theamount of scatter collected by the imaging optics. This is important forsamples with high scattering/reflection characteristics, such asmembrane-coated slides and glass surfaces.

Scattered and reflected signals are usually significant compared tofluorescence signals. As one anticipated use for the invention includesmembrane-coated and glass-type slides, angular illumination may bepreferred over co-axial illumination, unless rejection filters with goodblocking capabilities are used.

Angular illumination incidence can also be used to measure defocus. Onthe other hand, co-axial illumination with good rejection filters can behelpful for avoiding defocus effects in spectral filtering.

Dispersion Element

In order to set spectral filters at the CCD array, the optical signalcollected by the imaging optics needs to be spread (i.e., diffracted ordistributed) across the dimension of the CCD that is perpendicular tothe direction of the imaged line. The amount of spectral spread isdetermined by the desired spectral resolution. For purposes ofillustration, the spectral resolution can be set to an exemplary linearspread of 1 nm/pixel. This is equivalent to spreading the colors of theoptical signal with a 1 nm spread over each row of pixels or 400 nm over400 pixel rows.

This type of hyperspectral dispersion can be accomplished through avariety of methods, as discussed earlier. To keep the solution simple,only one dispersive element is used for the embodiment of a system asshown in FIG. 4. Two types of dispersive elements are considered here:Prisms and Transmission Gratings.

Prism: An equilateral prism is one of the most economical dispersivesolutions available; however prisms typically are used for applicationswith low spectral resolution, and they require angular adjustments,which complicate the mechanics. Also, prism placement in an imaging pathcan be critical as field curvature can be introduced when the prism isplaced in non-parallel field directions. These challenges may bedifficult to overcome with the dimensions (field size and spectralresolution) of some applications that are contemplated by the invention.

Transmission Grating: The use of transmission gratings simplifies theoverall mechanical mounting requirements. The operating parameters of atransmission grating are defined as shown in FIG. 15.

The condition for the first diffraction order of a transmission gratingis given by${\lambda = {{d \cdot \sin}\quad\beta_{1}}},{{x\quad(\lambda)} = {{D \cdot \tan}\quad\beta_{1}}},{{{and}\quad\frac{\Delta\lambda}{\Delta\beta}} = \frac{\lambda}{\tan\quad\beta_{1}}}$

where λ is the wavelength of light, D is the distance to the CCD, and dis the grating period, or groove spacing. In order to obtain a 1 nm per9 μm pixel wavelength spread and using a grating with 75 lines/mm, D=120mm and β₁ needs to be ˜3°. Such small deviation angle results in minimaldegradation of the image quality across the whole field. At these smallangles sin β₁≈tan β₁≈β₁,

and therefore${{x\quad(\lambda)} = {\frac{D}{\sqrt{( \frac{d}{\lambda} )^{2} - 1}} = {N_{\lambda} \times 9\mu\quad m}}},{{\Delta\lambda} = {( {\frac{d}{D} \times 9\mu\quad m} )\quad\Delta\quad N_{\lambda}}}$

the diffraction efficiency of a 75 line/mm grating obtained fromDiffraction Products, Inc. This can be optimized by designing the blazewavelength of the grating to match a desired wavelength of the systemfor maximum efficiency.

Imaging & Spectral Filtering

A fiber bundle with 50 μm diameter fiber, depicted in FIG. 17A, wasimaged to confirm that 5 μm feature sizes are detectable by the system.The fibers have a 45 μm core and 2.5 μm thick cladding. The non-guidinggaps in a bundle packed with these fibers are equal to 5 μm. Light wascoupled into the bundle, the output end was imaged by the imaging systemand, as shown in FIG. 17B, the gaps are clearly visible.

Scattering rejection: Scattering from an excitation laser can be muchstronger than the fluorescence level from the fluorophore, even if theillumination impinges on the target at an angle to avoid the collectionof specular reflection. A strong scattering signal can result in thesaturation of the detector, which would make the identification of thefluorescence part of the imaged spectrum impossible or extremelydifficult, especially in the case of membrane-coated slides. Therefore,in order to achieve acceptable results, scattering must be blocked or atleast reduced to a level that is comparable to the level of thefluorescence signal.

A rejection filter placed in the optical path, for example between theimaging optics and the dispersive element, can be used to reduce thescattering down to the fluorescence level. The rejection filter can bedesigned to reject >90% of the wavelength range of the excitation sourceand to pass >95% of the fluorescence range of the emission from theilluminated target. In this way the scattering level is reduced so thatit can be easily identified and thus eliminated when the binningspectral range is defined. The binning range is defined to include onlypure fluorescence and thus spectral ranges outside of the defined range,which include the scattering range, can be effectively forced to zero.This method may exhibit better results than the use of emission filtersto block the scattering spectral range to OD6 or better, since even withsuch optical filtering, a strong scattering signal can result in adetectable level that would interfere with the fluorescence detection.

This is a significant differentiation from all other conventionalspectral filtering techniques, which aim at reducing the amount ofdetectable optical signal outside the desired spectral bandwidth to alow level, such as to OD6 or better, but not to zero. According to thisaspect of the present invention, on the other hand, the signal leveloutside the desired range is forced to zero by excluding it during thebinning. This results in a significantly better optical signal-to-noiseperformance.

Spectral Calibration: In order to accurately and reliably identify thespectral locations and bandwidths of the desired filters, the pixel rownumbers of the chip need to be mapped to actual wavelength values. Thistask is simplified by the fact that in the 1 nm/pixel regime, thespectral spreading is linear. Therefore, only one source with a knownwavelength distribution will suffice. For example, the scattering froman excitation wavelength at 685 nm or 785 nm can be aligned to a desiredrow number and the corresponding wavelength assigned to that row number.For even higher accuracy, two known wavelengths can be usedsimultaneously to identify the corresponding row number and the per rowwavelength change. In accordance with the invention, a number ofmeasurements were taken with 685 nm and 785 nm laser diodes and a whitelight source having 600 nm, 650 nm, 700 nm, 750 nm, 800 nm, and 850 nmthin-film band-pass filters. The results confirmed that the spectralresolution is linear and is directly related to the distance between the75 lines/mm transmission grating and the CCD.

Imaging with Spectral Filtering: Once the rows of the CCD sub-frame aremapped to wavelength values and the positions corresponding to thedesired filter are identified for spectral binning, the imaged line isscanned across the target and a 2-D image is generated for each filterby combining the outputs from binned line corresponding to the samefilter together. The on-chip filtering operation, then, involvesprogramming the CCD to bin together the rows that correspond to purefluorescence, and to zeroing out the spectral range outside thepass-band by fast-dumping the rows that correspond to scattering.Alternatively, those rows may be masked to prevent them from receivingany light.

Two-Dimensional Scans: 2-D scans are obtained by translating a targetsample in a direction perpendicular to the imaged line and reading dataat a rate of 50 frames/second, i.e. 50 images/second. Experimental testsconfirmed good reduction of background noise, and much better noisereduction capability than thin-film bandpass filtering as done in someprior art apparatus.

Optional Enhancements

In a CCD, the charges that are collected by the photodiodes can becoupled together and moved around the array by proper application ofclock signals to the inputs of the array. There is thus a large amountof flexibility in how the charges are read out of the device. In themost basic and common read-out, the frame read-out begins by performinga vertical transfer. The vertical transfer moves each row of chargesdown to the next storage row. The final row in the array gets moved intoa horizontal register. Next, a horizontal transfer is performed to readeach pixel that is contained in the horizontal register out one at atime. When complete, the next vertical transfer is performed, and thiscontinues until each pixel has been read out of the CCD array.

For imaging a photograph, this is exactly what is desired. Forhyperspectral row imaging, a more complex technique can be used toimprove read-out time (frame rate), and collect and compress the amountof data that needs to be processed. The following sections discuss theimportant techniques that were used to achieve the desired performance.

Fast Dump: fast dump is a way to quickly remove signals from unneededrows of the CCD. If the Fast Dump Gate (FDG) is enabled prior to andduring the vertical transfer, the charge contained in the final row ofthe CCD is erased rather than added to the horizontal transfer register.This allows the next vertical transfer to take place without everperforming the horizontal transfer. This allows for a large savings inread time since the vertical transfer occurs much faster than the entirehorizontal transfer (10 μs versus 146 μs for a single row in the presentcase). This feature of the CCD is important to the overall speed of thesystem.

Electronic Shuttering: Electronic shuttering allows for a programmableamount of integration time per frame. The length of the integration timecan be either shorter or longer than the frame read-out time. Thisflexibility allows for adjusting the gain of the image. A longerintegration time proportionally boosts the gain.

Fast Sub-frame Rates: One goal of the present invention was to obtain arate of 50 frames per second (fps) collected from the CCD. This rate isrequired to scan 75 mm at 5 μm resolution (15,000 lines), in 5 minutes.The Fast Dump feature was helpful, but the implementation at the time oftesting was not sufficient to obtain the desired goal. For example, ifit is desired to read four 50 nm wide spectral windows using dual analogoutputs, it would take (4 windows×50rows/window×78 μs/row)+(2520 rows×10μs/row)=40.8 ms, which is only 24.5 fps. 2520 is the number of unusedrows in the CCD array, which are “fast dumped” at a rate of 10 μs/row.Since all pixels are active, the entire array has to be read out inorder to clear the CCD storage array each frame. Two techniques wereused to dramatically increase the frame read-out time: masking andspectral binning.

Masking: In the example case above, most of the rows of the array arenot being used for useful data acquisition. The time required to fastdump all those unused rows was prohibitory to achieving the desired 50fps rate. From the previous example, 25.2 ms of the total frameacquisition time of 40.8 ms was required just to purge the unused areaof the CCD array. By masking the unused area of the array, the maskedpixels are no longer subject to interaction with light. Therefore whilesignals from the masked rows would be added to the read rows, the maskedrow signals are near zero (there is some slight dark noise associatedwith the masked rows, but it is very small in comparison to other noisesources in the system). By masking off the unused rows, a huge timeadvantage is achieved. In the same example, if only 450 rows are readout (200 unmasked rows+250 rows for mask alignment calibration (whichcan be fast-dumped)), then the total frame readout time is 4 unmaskedwindows×50 rows/window×78 μs/row+250 mask alignment (fast dump) rows×10μs/row=18.1 ms. This rate is 55.2 fps and more than achieves the desired50 fps goal. An alternative to masking off part of a large CCD such asused in this testing, is to use a rectangular CCD with a length thatmatches the desired spectral window, e.g., 4000×450 format for 500nm-950 nm at a resolution of 1 nm/row.

Spectral Binning: Multiple rows of a CCD can be added together on theCCD chip before being read out, as explained above. This is accomplishedby performing two or more vertical transfers without performing ahorizontal transfer. While similar to the Fast Dump technique, thedifference is that the FDG line is not activated during the verticaltransfer. In this case, the charge from the final row of the CCD isadded to the contents of the horizontal transfer register. In theconfiguration tested, this allows any number of sequential wavelengthrows to be combined on-chip. This allows the creation of custom spectralfilters just by changing which portions of the frame are binned andwhich are fast dumped.

This method represents a significant advantage. Because the noise of theCCD is dominated by read noise at high frame rates, a significantpenalty is paid each time a pixel is read out. By combining the smallpixel signals on the CCD before reading them out, the read-noise penaltyis paid only once per spectral window. The trade-off in doing this isthat the horizontal transfer register can only hold the charge of twofull-value pixels. For a 30 to 50 nm spectral filter, 30 to 50 rowswould be added together. If the fluorescence signal is large, thenadding 30 to 50 pixels together into a horizontal pixel that can holdonly as much as 2 full pixels will very likely saturate the pixel. Thedynamic range of the horizontal register is however still limited by theresolution of the A/D converter used. The positive side of this is thatthe Limit of Detection (LD) can be improved because the entire signal isadded together without paying the read-noise penalty. Also, theflexibility offered by this spectral binning method allows forsub-dividing the spectrum for each filter into sections with a varyingnumber of lines binned so as to avoid saturating the pixels; This way,good LD is achieved as well as maintaining good Dynamic Range (DR). TheLD also can be improved by increasing the S/N ratio at low signallevels. This can be achieved either by increasing the signal leveland/or reducing the noise. The signal can be increased by increasing thepower of the illumination source and/or lengthening the CCD integrationtime. At the same time, the noise can be reduced by reducing theread-out noise and optical background noise. This may be accomplished bycontrolling the rejection filter specifications as well as the spectralbinning performance.

There are two additional significant benefits to this method. First,adding two rows together on the CCD takes only 10 μs, whereas readingout a row to be added externally would take 78 μs. So the total frameread-out time is much faster. Secondly, since now only one row perfilter window is being read, there is no post-acquisition processing toexecute. Post processing each 1 nm row and combining them into a totalspectral window would require the use of a fast DSP or FPGA chip, and arelatively large amount of fast memory. As an example of the timesavings, assume the same 4 windows each 50 nm wide, with a total of 450rows being read out per frame. With on-chip spectral binning, this wouldtake 4×(49×10 μs)+(4'78 μs)+(250×10 μs)=4.77 ms. Without on-chipspectral binning, this would take 4×(50×78 μs)+(250×10 μs)=18.1 ms. Thisis an almost 4× improvement in the frame read-out time alone, with nopost-acquisition processing being required.

In summary, a new spectral filtering apparatus and method has beenpresented, which enables the functions of hyperspectral/multi-spectralimaging to be performed with increased speed, simplicity, andperformance that are required for commercial products. Some of theadvantages of the invention are:

1. Speed: On-chip spectral filtering achieves minimal data readoutrequirements and virtually no post-acquisition processing;

2. Filtering flexibility: The number of spectral filters and theirparameters are implemented by programming the CCD and can be changed atwill;

3. Parallel detection of overlapping fluorescence: The same CCD can beused to detect emissions that span the same spectral range at the sametime. This can be achieved by simply spatially offsetting theircorresponding excitations;

4. Improved scattering rejection: Rejection filters are used to reducethe amount of scattering to the level of fluorescence, in order to avoidsaturating the CCD. Any remaining amount of scattering is then forced tozero by the filtering operation. This results in an improved S/N;

5. Reduced noise. The on-chip binning prior to the readoutpre-amplification results in a significant reduction in total readnoise;

6. Improved Limit of Detection: This results primarily from binning weaksignals together prior to the addition of read noise;

7. Increased Dynamic Range: Fast shuttering and fast detection allow formore than one exposure per each line image. This results in an increaseddynamic range capability. Also, the control of integration time improvesthe limit of detection and dynamic range;

8. Depth information: Angular illumination allows for measuring theheight variations of the imaged surface and therefore provide for a wayto correct for defocus. The same concept can be applied to produce 3-Dimages;

9. Confocality: Angular illumination with array detection can also beused to apply confocal aperture operations and therefore obtain confocalimages; and

10. Extended range of applications: This method can be used in manyapplications that use multi-spectral or hyperspectral detection such asin microscopy, remote sensing, space imaging, and military focal planearray applications.

Additionally, while the invention has been described with respect to theuse of CCD arrays, it is contemplated that other photosensitivetechnologies also may be used with the concepts of the invention,including APD, CMOS and PMT.

Defocus correction is also possible with the invention, wherein a zeroorder line would be used for focusing, and higher order regions used forspectral content detection. Focus correction could be carried out bylinking the focus to spectral information, either pixel-by-pixel, or bytaking the average of the offset value.

1. A spectral filter, comprising: a photosensitive array having aplurality of photosensitive elements, at least a subsection ofphotosensitive elements of said array along one direction thereof beingconfigured such that each element of said subsection along said onedirection receives light of a different wavelength range of acharacteristic spectrum and produces an electrical signal correspondingthereto, said subsection being configured to combine signals of at leasttwo photosensitive elements and to output said combined signal as ameasure of the optical energy within a bandwidth of interest of saidcharacteristic spectrum.
 2. The spectral filter of claim 1, wherein saidarray is a CCD array.
 3. The spectral filter of claim 1, wherein saidarray is a CMOS array.
 4. The spectral filter of claim 1, wherein saidarray is a PhotoDiode (PD) array.
 5. The spectral filter of claim 1,wherein said array is an Avalanche PhotoDiode (APD) array.
 6. Thespectral filter of claim 1, wherein said array is a PMT array.
 7. Thespectral filter of claim 1, further comprising a dispersive element forspreading said characteristic spectrum across a dimension of saidphotosensitive array corresponding to said one direction.
 8. Thespectral filter of claim 7, wherein said photosensitive array is an areaarray, and said one direction corresponds to different lines ofphotosensitive elements.
 9. The spectral filter of claim 7, wherein saidphotosensitive array is a line array, and said one direction correspondsto different photosensitive elements.
 10. The spectral filter of claim7, wherein said dispersive element is a transmission grating.
 11. Thespectral filter of claim 7, wherein said dispersive element is a prism.12. A system for spectral imaging of a target, comprising: aphotosensitive array having a plurality of photosensitive elements; anillumination source for excitation of said target; imaging optics thatfocuses and spreads light emitted from said target in response toexcitation from said illumination source over said photosensitive array,such that different subsections of photosensitive elements of said arrayreceive light of a different wavelength range of a characteristicspectrum of said target; and output electronics coupled to saidphotosensitive array and receiving electrical signals from saidphotosensitive elements, combining signals of at least twophotosensitive elements in a subsection and outputting said combinedsignal as a measure of the optical energy within a bandwidth of interestof said characteristic spectrum.
 13. The system of claim 12, whereinsaid array is a CCD array.
 14. The system of claim 12, wherein saidillumination source is a laser source.
 15. The system of claim 14,wherein said laser source comprises a plurality of lasers, eachproducing light of a different wavelength.
 16. The system of claim 12,wherein said imaging optics comprises a focusing lens and a dispersiveelement.
 17. The system of claim 12, wherein said dispersive element isa transmission grating.
 18. The system of claim 12, wherein saiddispersive element is a prism.
 19. The system of claim 12, wherein saidoutput electronics is programmable to select different combinations ofphotosensitive elements for signal combination.
 20. The system of claim19, wherein said photosensitive array is an area array, and said outputelectronics is programmable to select different lines of photosensitiveelements for signal combination.
 21. The system of claim 19, whereinsaid photosensitive array is a line array, and said output electronicsis programmable to select different photosensitive elements for signalcombination.
 22. A method of performing spectral filtering using aphotosensitive array having a plurality of photosensitive elements,comprising the steps of: calibrating said array such that each elementof said photosensitive array along one direction thereof corresponds toa different wavelength range of a characteristic spectrum projected ontosaid array; configuring a read-out process of said array to combinesignals of at least two photosensitive elements along said onedirection, generated in response to illumination by said characteristicspectrum, as a measure of the optical energy within a bandwidth ofinterest of said spectrum.
 23. The method of claim 22, wherein saidarray is a CCD array.
 24. The method of claim 22, wherein the step ofcalibrating comprises the step of positioning said array with respect toa target emitting said characteristic spectrum and with respect toimaging optics positioned between said array and said target.
 25. Themethod of claim 22, wherein the step of configuring comprises the stepof programming output electronics with respect to generation of timingsignals applied to an output register coupled to receive signals fromsaid photosensitive elements.
 26. The method of claim 23 wherein saidCCD array is an area array, and further comprising the step of using azero order line of said array for focusing of said characteristicspectrum.